Using blind analysis for software engineering experiments

Context: In recent years there has been growing concern about conflicting experimental results in empirical software engineering. This has been paralleled by awareness of how bias can impact research results. Objective: To explore the practicalities of blind analysis of experimental results to reduce bias. Method : We apply blind analysis to a real software engineering experiment that compares three feature weighting approaches with a na ̈ıve benchmark (sample mean) to the Finnish software effort data set. We use this experiment as an example to explore blind analysis as a method to reduce researcher bias. Results: Our experience shows that blinding can be a relatively straightforward procedure. We also highlight various statistical analysis decisions which ought not be guided by the hunt for statistical significance and show that results can be inverted merely through a seemingly inconsequential statistical nicety (i.e., the degree of trimming). Conclusion: Whilst there are minor challenges and some limits to the degree of blinding possible, blind analysis is a very practical and easy to implement method that supports more objective analysis of experimental results. Therefore we argue that blind analysis should be the norm for analysing software engineering experiments

Scientific applications of radio and radar tracking in the space program Conference proceedings

Radar and radio tracking applications in space progra

Frequentist and Bayesian measures of confidence via multiscale bootstrap for testing three regions

A new computation method of frequentist $p$-values and Bayesian posterior probabilities based on the bootstrap probability is discussed for the multivariate normal model with unknown expectation parameter vector. The null hypothesis is represented as an arbitrary-shaped region. We introduce new parametric models for the scaling-law of bootstrap probability so that the multiscale bootstrap method, which was designed for one-sided test, can also computes confidence measures of two-sided test, extending applicability to a wider class of hypotheses. Parameter estimation is improved by the two-step multiscale bootstrap and also by including higher-order terms. Model selection is important not only as a motivating application of our method, but also as an essential ingredient in the method. A compromise between frequentist and Bayesian is attempted by showing that the Bayesian posterior probability with an noninformative prior is interpreted as a frequentist $p$-value of ``zero-sided'' test

Universality Class of One-Dimensional Directed Sandpile Models

A general n-state directed `sandpile' model is introduced. The stationary properties of the n-state model are derived for n < infty, and analytical arguments based on a central limit theorem show that the model belongs to the universality class of the totally asymmetric Oslo model, with a crossover to uncorrelated branching process behavior for small system sizes. Hence, the central limit theorem allows us to identify the existence of a large universality class of one-dimensional directed sandpile models.Comment: 4 pages, 2 figure

Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation

Image segmentation is a fundamental problem in biomedical image analysis. Recent advances in deep learning have achieved promising results on many biomedical image segmentation benchmarks. However, due to large variations in biomedical images (different modalities, image settings, objects, noise, etc), to utilize deep learning on a new application, it usually needs a new set of training data. This can incur a great deal of annotation effort and cost, because only biomedical experts can annotate effectively, and often there are too many instances in images (e.g., cells) to annotate. In this paper, we aim to address the following question: With limited effort (e.g., time) for annotation, what instances should be annotated in order to attain the best performance? We present a deep active learning framework that combines fully convolutional network (FCN) and active learning to significantly reduce annotation effort by making judicious suggestions on the most effective annotation areas. We utilize uncertainty and similarity information provided by FCN and formulate a generalized version of the maximum set cover problem to determine the most representative and uncertain areas for annotation. Extensive experiments using the 2015 MICCAI Gland Challenge dataset and a lymph node ultrasound image segmentation dataset show that, using annotation suggestions by our method, state-of-the-art segmentation performance can be achieved by using only 50% of training data.Comment: Accepted at MICCAI 201

Some Aspects of Measurement Error in Linear Regression of Astronomical Data

I describe a Bayesian method to account for measurement errors in linear regression of astronomical data. The method allows for heteroscedastic and possibly correlated measurement errors, and intrinsic scatter in the regression relationship. The method is based on deriving a likelihood function for the measured data, and I focus on the case when the intrinsic distribution of the independent variables can be approximated using a mixture of Gaussians. I generalize the method to incorporate multiple independent variables, non-detections, and selection effects (e.g., Malmquist bias). A Gibbs sampler is described for simulating random draws from the probability distribution of the parameters, given the observed data. I use simulation to compare the method with other common estimators. The simulations illustrate that the Gaussian mixture model outperforms other common estimators and can effectively give constraints on the regression parameters, even when the measurement errors dominate the observed scatter, source detection fraction is low, or the intrinsic distribution of the independent variables is not a mixture of Gaussians. I conclude by using this method to fit the X-ray spectral slope as a function of Eddington ratio using a sample of 39 z < 0.8 radio-quiet quasars. I confirm the correlation seen by other authors between the radio-quiet quasar X-ray spectral slope and the Eddington ratio, where the X-ray spectral slope softens as the Eddington ratio increases.Comment: 39 pages, 11 figures, 1 table, accepted by ApJ. IDL routines (linmix_err.pro) for performing the Markov Chain Monte Carlo are available at the IDL astronomy user's library, http://idlastro.gsfc.nasa.gov/homepage.htm

Comprehensive quantum Monte Carlo study of the quantum critical points in planar dimerized/quadrumerized Heisenberg models

We study two planar square lattice Heisenberg models with explicit dimerization or quadrumerization of the couplings in the form of ladder and plaquette arrangements. We investigate the quantum critical points of those models by means of (stochastic series expansion) quantum Monte Carlo simulations as a function of the coupling ratio $\alpha = J^\prime/J$. The critical point of the order-disorder quantum phase transition in the ladder model is determined as $\alpha_\mathrm{c} = 1.9096(2)$ improving on previous studies. For the plaquette model we obtain $\alpha_\mathrm{c} = 1.8230(2)$ establishing a first benchmark for this model from quantum Monte Carlo simulations. Based on those values we give further convincing evidence that the models are in the three-dimensional (3D) classical Heisenberg universality class. The results of this contribution shall be useful as references for future investigations on planar Heisenberg models such as concerning the influence of non-magnetic impurities at the quantum critical point.Comment: 10+ pages, 7 figures, 4 table

Nonparametric Regression using the Concept of Minimum Energy

It has recently been shown that an unbinned distance-based statistic, the energy, can be used to construct an extremely powerful nonparametric multivariate two sample goodness-of-fit test. An extension to this method that makes it possible to perform nonparametric regression using multiple multivariate data sets is presented in this paper. The technique, which is based on the concept of minimizing the energy of the system, permits determination of parameters of interest without the need for parametric expressions of the parent distributions of the data sets. The application and performance of this new method is discussed in the context of some simple example analyses.Comment: 10 pages, 4 figure

Critical behavior of the three-dimensional bond-diluted Ising spin glass: Finite-size scaling functions and Universality

We study the three-dimensional (3D) bond-diluted Edwards-Anderson (EA) model with binary interactions at a bond occupation of 45% by Monte Carlo (MC) simulations. Using an efficient cluster MC algorithm we are able to determine the universal finite-size scaling (FSS) functions and the critical exponents with high statistical accuracy. We observe small corrections to scaling for the measured observables. The critical quantities and the FSS functions indicate clearly that the bond-diluted model for dilutions above the critical dilution p*, at which a spin glass (SG) phase appears, lies in the same universality class as the 3D undiluted EA model with binary interactions. A comparison with the FSS functions of the 3D site-diluted EA model with Gaussian interactions at a site occupation of 62.5% gives very strong evidence for the universality of the SG transition in the 3D EA model.Comment: Revised version. 10 pages, 9 figures, 2 table

Collective motion of self-propelled particles interacting without cohesion

We present a comprehensive study of Vicsek-style self-propelled particle models in two and three space dimensions. The onset of collective motion in such stochastic models with only local alignment interactions is studied in detail and shown to be discontinuous (first-order like). The properties of the ordered, collectively moving phase are investigated. In a large domain of parameter space including the transition region, well-defined high-density and high-order propagating solitary structures are shown to dominate the dynamics. Far enough from the transition region, on the other hand, these objects are not present. A statistically-homogeneous ordered phase is then observed, which is characterized by anomalously-strong density fluctuations, superdiffusion, and strong intermittency.Comment: Submitted to Physical Review